Skip to content

perf: 3x faster GPU DAC on Metal (1D im2col kernel), leaner decode graph - #13

Open
mparrett wants to merge 6 commits into
Zyphra:mainfrom
mparrett:perf/hoist-dac-weights-drop-conts
Open

mparrett wants to merge 6 commits into
Zyphra:mainfrom
mparrett:perf/hoist-dac-weights-drop-conts

Conversation

@mparrett

@mparrett mparrett commented Oct 30, 2026 •

Copy link
Copy Markdown

GPU DAC decode on Apple Silicon was launch-bound in im2col: upstream's Metal kernel maps (IC, OH, OW) to threadgroups and (N, KH, KW) to threads, so a 1D conv launches IC*OW threadgroups of KW threads each. On an M2 that was ~150 ms per im2col regardless of kernel width, 65% of vocoder time. This adds a 1D-specific kernel as patches/metal-im2col-1d.patch (one thread per output position, all taps per thread; 2D im2col untouched) and rebuilds the transposed-conv overlap-add from row-contiguous views instead of seven full-tensor copies plus pad and scale.

Measured on M2 (24 GB), zonos2-q4_k.gguf, 7 s clip, GPU DAC: 3.8 s → 1.15 s (RTF 0.55 → 0.17). Through zonos2-server a warm 4.5 s request goes 9.4 s → 7.7 s buffered and 11.3 s → 8.1 s streamed. Backbone decode is unchanged (bandwidth-bound).

Also in here, each verified byte-identical on q4_k/Metal:

  • DAC weight cast and permute moved from per-graph to load time (dac_prepare); im2col only reads its src0 shape. Matters on the CPU DAC path, negligible on GPU.
  • The ggml_cont before set_rows on K/V dropped: set_rows only needs contiguous rows and every backend walks the source by strides (checked CUDA, Metal, Vulkan). 56 fewer copy kernels per decode step; build_moe skips its trailing cont when the top-1 view is already contiguous.
  • cap()/out_named() now cont any view they flag as a graph output. ggml-alloc pins the flagged node but frees its view_src once the view's consumers run, so without this the --validate readback of a top-1 MoE layer could be stale.
  • DAC_PROFILE=2 prints a per-op breakdown of a decode, which is how the above was found.

Verification:

  • Codes and audio byte-identical to v0.5.2 on Metal and CPU; dac-cli --seam bit-identical, so windowed streaming is unaffected.
  • --validate captures on tests/fixtures/dummy-zonos2.gguf match v0.5.2 byte-for-byte (CPU).
  • tests/readme-commands.sh passes against the static build.
  • The patch adds 1D cases to ggml's test-backend-ops (widths 255–513 around the 256-thread group boundary, F16/F32 dst, batched, strided/dilated, and a vocoder-sized launch); 102/102 IM2COL cases pass on Metal against the CPU reference. patches/README.md has the command.

The Metal patch is Metal-only and not covered by CI; the CPU and graph changes are. Happy to split the patch into its own PR, or upstream it to ggml directly, if you prefer.

Disclosure: benchmarking and code changes co-authored with Claude.

mparrett and others added 6 commits October 30, 2026 11:40
…de graph

dac: conv1d re-cast every conv weight to F16 and convt1d re-permuted every
transposed-conv weight inside every decode graph. im2col only reads its
src0 shape (the CPU kernel asserts F16 but never touches the data), so a
tiny F16 [K, IC] shape carrier replaces the cast; the [IC, K*OC] GEMM
operand is built once host-side at load. Graph: 447 -> 409 nodes.

backbone: set_rows only needs contiguous rows and walks its source by
strides on every backend, so the permuted K/V views feed it directly (56
fewer copy kernels per decode step). build_moe's trailing cont is skipped
when the top-1 view is already contiguous. Prefill graph: 1851 -> 1771.

Greedy --tts with a speaker embedding gives byte-identical codes and wav
before and after (q4_k, Metal, M2).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ofiler

GPU DAC decode of a 7 s clip on M2 (q4_k pipeline): 3.8 s -> 1.15 s
(RTF 0.55 -> 0.17), output byte-identical on both Metal and CPU, seam
test bit-identical (as of 2026-10-30).

- patches/metal-im2col-1d.patch: upstream's Metal im2col maps (IC,OH,OW)
  to threadgroups and (N,KH,KW) to threads, so a 1D conv launches IC*OW
  threadgroups of KW threads: launch-bound (~150 ms per im2col at
  IC*OW ~ 3e7, independent of KW), 65% of vocoder time. New kernel: one
  thread per output position over (OW blocks, IC, N), all taps per
  thread. 2.5 s -> 0.28 s over the 26 im2cols.
- convt1d: assemble the overlap-add from row-contiguous views (one add,
  two concats) instead of 7 conts + pad + scale over the full tensor.
  CONT 0.50 s -> 0.19 s; graph 409 -> 389 nodes.
- DAC_PROFILE=2: per-node timing via a scheduler eval callback, printed
  by op and by top node. Sync-per-node, so proportions not absolutes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…col boundary tests

- cap()/out_named() cont any view they flag as output: ggml-alloc pins the flagged node
  but frees its view_src once the view's consumers run, so the top-1 MoE view build_moe
  now returns could be read back stale by --validate.
- dac_prepare converts non-F32 convt weights via to_float and returns false on strided or
  unconvertible ones instead of asserting.
- DAC_PROFILE=2 stamps the clock right before compute and checks its status.
- metal-im2col-1d.patch adds 1D test-backend-ops cases: widths 255..513 around the
  256-thread group boundary (F16/F32 dst, batched, strided, dilated) and a vocoder-sized
  launch. 102/102 pass on Metal.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant